On Sequentially Normalized Maximum Likelihood Models

The important normalized maximum likelihood (NML) distribution is obtained via a normalization over all sequences of given length. It has two short-comings: the resulting model is usually not a random process, and in many cases, the normalizing integral or sum is hard to compute. In contrast, the recently proposed sequentially normalized maximum likelihood (SNML) models always comprise a random process and are often much easier to compute. We present some results on SNML type models in the Markovian and linear–Gaussian model classes. In the linear–Gaussian case, the resulting sequentially normalized least squares (SNLS) model is particularly interesting. The associated sequentially minimized squared deviations are smaller than both the usual least squares and the squared prediction errors used in the so called predictive least squares (PLS) criterion. The SNLS model is asymptotically optimal within the given class of distributions by reaching the lower bound on the logarithmic prediction errors, given by the stochastic complexity, up to lower-order terms.